Fluorescence microscopy inspection systems, apparatus and methods

ABSTRACT

A fluorescence microscopy inspection system includes light sources able to emit light that causes a specimen to fluoresce and light that does not cause a specimen to fluoresce. The emitted light is directed through one or more filters and objective channels towards a specimen. A ring of lights projects light at the specimen at an oblique angle through a darkfield channel. One of the filters may modify the light to match a predetermined bandgap energy associated with the specimen and another filter may filter wavelengths of light reflected from the specimen and to a camera. The camera may produce an image from the received light and specimen classification and feature analysis may be performed on the image.

CROSS-REFERENCE

This application is related to and claims priority under 35 U.S.C. §119(e) from U.S. Patent Appl. No. 62/802,246 entitled “INCOHERENTULTRAVIOLET LIGHT AUTOMATIC MICROSCOPY INSPECTION SYSTEMS, APPARATUS ANDMETHODS” and filed Feb. 7, 2019 and U.S. Patent Appl. No. 62/836,206,entitled “PHOTOLUMINESCENCE IMAGING FOR DETECTION AND CLASSIFICATION OFSTACKING FAULTS IN 4H—SiC” and filed Apr. 19, 2019, the entire contentsof each of which is incorporated herein by reference for all purposes.

TECHNICAL FIELD

The present disclosure generally relates to fluorescence microscopyinspection systems, apparatus and methods using incoherent illuminationtechniques. More particularly, embodiments of the present inventionrelate to fluorescence microscopy inspection systems that can provideincoherent light of variable wavelengths targeted to excite specificlayers of a specimen or materials contained in a specimen andautomatically detect features of a specimen from the resultingfluorescence caused by the absorption of light or other electromagneticradiation.

BACKGROUND

Projecting non-visible light at a specimen and capturing the resultingfluorescence/photoluminescence emitted by a specimen can provideimportant information about the quantity, type, location and morphologyof features on a specimen. Further, certain features of a specimen, suchas the purity or structural imperfections of the specimen, among others,may only be observed using non-visible illumination. Specimens asunderstood by a person of ordinary skill in the art refer to an articleof examination (e.g., a wafer or a biological slide) and features referto observable characteristics of a specimen, including abnormalitiesand/or defects. Features can include but are not limited to: circuits,circuit board components, biological cells, tissue, defects (e.g.,impurities, structural imperfections, irregularities, stacking faults,contaminants, crystallographic defects, scratches, dust, fingerprints).

Note, the term fluorescence (FL) as used herein includesphotoluminescense, which is commonly associated with light emissionsfrom semiconductor materials. Non-visible light refers to the region ofthe electroagnetic spectrum with a wavelength between 10 and 400nanometers (nm) (i.e., the region between visible light and X-rays). Insome embodiments, for example, light wavelengths in the range of 200 nmto 400 nm, 300 nm to 400 nm, and/or any other suitable wavelengths canbe selected. Moreover, the light wavelength required to excite aspecimen and cause fluoresencese by a specimen from the absorption oflight or other electromagnetic radiation is not restricted to thewavelength range between 10 nm to 400 nm, but, in some embodiments, canbe selected in a range above 400 nm to provide the desired excitation toa specimen, as explained herein. Coherent light refers to particles oflight energy that have the same frequency and its waves are in phasewith one another. In contrast, the particles of light energy ofincoherent light do not have the same frequency and its waves are not inphase with one another.

While coherent light sources (e.g., lasers) are commonly used forspecimen fluorescence, such light sources are not ideal for detectinglarge features or for use with certain types of specimens (e.g.,patterned wafers). Incoherent light sources, on the other hand, arebetter suited for detecting a greater range of features (including largefeatures and features on patterned wafers). Moreover, coherent lightsources illuminate only a small portion of a field of view, whereasincoherent light illuminates the entire field of view, making it moresuitable for creating specimen feature maps. Specimen feature mapsclassify features on a specimen and specify their location. Note, theterm field of view as understood by a person of ordinary skill in theart refers to an area of examination that is captured at once by animage sensor. Further, a person of ordinary skill in the art willreadily understand that the terms field of view and image are usedinterchangeably herein.

Accordingly, new fluorescence microscopy inspection mechanisms usingincoherent illumination techniques are desirable to excite specificlayers of a specimen or materials contained in a specimen to cause themto fluoresce and to automatically detect features of a specimen from theresulting fluorescence. Moreover, it is also desirable for the samemechanisms to inspect features of a specimen using illuminationtechniques that do not cause fluorescence.

SUMMARY

In one example, a system includes a frame, one or more incoherent lightsources connected to the frame and configured to emit at least a firstwavelength of light that will cause a specimen to fluoresce and a secondwavelength of light that will not cause a specimen to fluoresce, whereinthe emitted light is configured to be directed to the specimen, anexcitation filter connected to the frame and configured to filter lightfrom the one or more light sources, wherein the filtered light isconfigured to match a predetermined bandgap energy associated with thespecimen, an objective connected to the frame, the objective comprisinga brightfield channel and a darkfield channel, a slider connected to theframe and positioned along a lightpath between the objective and the oneor more incoherent light sources, wherein the slider includes at leastone configuration configured to transmit light along the lightpath to atleast a darkfield channel configured to direct light to the specimen atan oblique angle, and an emission filter connected to the frame andconfigured for filtering selected wavelengths of light reflected fromthe specimen to a receiving camera.

In some examples, the at least one configuration of the slider isconfigured to transmit light to both the brightfield channel and thedarkfield channel.

In some examples, the system further includes a nosepiece connected tothe frame, wherein the objective is connected to the nosepiece via anattachment, and a darkfield insert fastened to the attachment andpositioned above the darkfield channel of the objective, the darkfieldinsert including a ring of lights configured to project light at thespecimen at an oblique angle.

In some examples, the slider is a filter slider connected to the frameand positioned below the darkfield insert, the filter slider configuredto provide multiple types of excitation filters, and one or moreadditional emission filters for one or more of the brightfield channelor the darkfield channel.

In some examples, the system further includes at least a second cameraand the emitted light includes visible light and non-visible lightdirected to respective cameras.

In some examples, the system includes one or more additional cameraconnected to the frame, each additional camera configured to receiverespective unique wavelengths of light.

In some examples, the system further includes one or more processors,and a memory storing instructions that, when executed by the one or moreprocessors, cause the one or more processors to receive image data fromthe receiving camera, the image data based on the directed light fromthe specimen, classify the specimen with a trained classifier based onthe received image data, retrieve stored system configurationsassociated with the classification of the specimen, and apply the systemconfigurations to one or more of the light sources, excitation filter,emission filter, or receiving camera.

In some examples, the memory stores further instructions to receiveadditional image data from the receiving camera, the additional imagedata received after the system configurations have been applied,identify specimen defects with an image data model based on the receivedadditional image data, and generate a feature map based on the specimendefects.

In some examples, the one or more incoherent light sources furtherincludes a first light source connected to the frame and configured toemit reflected light from the one or more incoherent light sources tothe specimen, and an additional light source attached to the frame belowthe specimen and configured to increase the intensity of light on thespecimen by emitting light directed at the specimen simultaneously tothe light emitted by the one or first incoherent light source.

In some examples, the system further includes a beam splitter connectedto the frame and configured to direct emitted light towards thespecimen.

In one example, a method includes emitting from one or more incoherentlight sources a first wavelength of light that causes a specimen tofluoresce and a second wavelength of light that does not cause thespecimen to fluoresce, wherein the emitted light is directed to thespecimen, filtering the emitted light through an excitation filter, thefiltered light matching a predetermined bandgap energy, transmitting theemitted light through a slider to the specimen via a darkfield channelof an objective at an oblique angle, and directing light reflected fromthe specimen to a receiving camera, the reflected light in response tothe directed filtered light wherein the directed light reflected fromthe specimen comprises selected wavelengths.

In some examples, the method further includes transmitting the filteredlight to the specimen through the slider to a brightfield channel of theobjective.

In some examples, the method includes a darkfield insert, positionedabove the darkfield channel of the objective, including a ring oflights, emitting light to the specimen at an oblique angle via thedarkfield channel of the objective.

In some examples, the emitted light includes visible light andnon-visible light, and the method further includes receiving, by asecond camera, at least a portion of the directed light reflected fromthe specimen.

In some examples, the method includes receiving, by one or moreadditional cameras, unique wavelengths of the light reflected from thespecimen.

In some examples, the method includes receiving image data from thereceiving camera, the image data based on the directed light reflectedfrom the specimen, classifying the specimen with a trained classifierbased on the received image data, retrieving stored systemconfigurations associated with the classification of the specimen, andapplying the system configurations to one or more of the light sources,excitation filter, emission filter, or receiving camera.

In some examples, the method includes receiving additional image datafrom the receiving camera, the additional image data received after thesystem configurations have been applied, identifying specimen defectswith an image data model based on the received additional image data,and generating a feature map based on the specimen defects.

In some examples, the method further includes emitting a first lightfrom a first light source of the one or more incoherent light sourcestoward the specimen, and increasing the intensity of light on thespecimen, by emitting a second light directed at the specimen from anadditional light source of the one or more incoherent light sources,from below the specimen, wherein the additional light is emittedsimultaneously to the light emitted by the first light source of the oneor more incoherent light sources.

In some examples, the method further includes directing the emittedlight towards the specimen with a beam splitter.

In one example, an apparatus includes one or more incoherent lightsources configured to emit at least a first wavelength of light thatwill cause a specimen to fluoresce and a second wavelength of light thatwill not cause the specimen to fluoresce, wherein the emitted light isconfigured to be directed to the specimen, an excitation filterconfigured to filter light from the one or more light sources, whereinthe filtered light is configured to match a predetermined bandgap energyassociated with the specimen, an objective including a brightfieldchannel and a darkfield channel, a nosepiece connected to the objectivevia an attachment, a darkfield insert fastened to the attachment andpositioned above the darkfield channel of the objective, the darkfieldinsert including a ring of lights configured to project light at thespecimen at an oblique angle, an emission filter configured forfiltering selected wavelengths of reflected light from the specimen to areceiving camera, one or more processors, and a memory storinginstructions that, when executed by the one or more processors, causethe one or more processors to receive image data from the receivingcamera, the image data based on the directed light reflected from thespecimen, classify the specimen with a trained classifier based on thereceived image data, retrieve stored system configurations associatedwith the classification of the specimen, apply the system configurationsto one or more of the light sources, excitation filter, emission filter,or receiving camera, receive additional image data from the receivingcamera, the additional image data received after the systemconfigurations have been applied, identify specimen defects with animage data model based on the received additional image data, andgenerate a feature map based on the specimen defects.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the disclosure can be obtained, a moreparticular description of the principles briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only exemplary embodiments of the disclosure and are nottherefore to be considered to be limiting in their scope, the principlesherein are described and explained with additional specificity anddetail through the use of the accompanying drawings in which:

FIGS. 1A and 1B illustrate examples of fluorescence microscopyinspection systems, according to some aspects of the disclosedtechnology.

FIG. 2 shows an example embodiment of a fluorescence microscopyinspection systems that includes two imaging devices.

FIGS. 3A and 3B show example embodiments of a brightfield/darkfieldslider in the light path of a fluorescence microscopy inspection system.

FIGS. 3C, 4A and 4B show example embodiments of a brightfield/darkfieldslider.

FIG. 5A shows an example embodiment of a fluorescence microscopyinspection system with a cylinder attachment that includes a darkfieldinsert.

FIG. 5B shows an example nosepiece with a cylinder attachment thatincludes a darkfield insert.

FIG. 6A shows an example embodiment of a darkfield insert.

FIG. 6B shows an example embodiment of a cylinder attachment.

FIG. 6C shows an example embodiment of a filter slider.

FIG. 7A shows at a high level, an example method for illuminating aspecimen using a FM inspection system, according to some aspects of thedisclosed technology.

FIG. 7B illustrates steps of an example process for identifying aspecimen classification and automatically adjusting the light sourcesand filters for FM inspection system, according to some aspects of thedisclosed technology.

FIG. 7C illustrates steps of an example process for automaticallyidentifying and/or classifying specimen defects, according to someaspects of the disclosed technology.

FIG. 8 shows the general configuration of an embodiment of computeranalysis system, in accordance with some embodiments of the disclosedsubject matter.

FIG. 9 shows an image processing algorithm that is first trained withtraining data so that an image processing module can identify a specimenand features on a specimen.

FIG. 10 illustrates an example classification method using convolutionalneural networks (CNNs), according to some aspects of the disclosedtechnology.

DETAILED DESCRIPTION

In accordance with some embodiments of the disclosed subject matter,mechanisms (which can include systems, methods, devices, apparatuses,etc.) for fluorescence microscopy inspection using incoherentillumination techniques to excite specific layers of a specimen ormaterials contained in a specimen to cause them to fluoresce and toautomatically detect features of a specimen from the resultingfluorescence are provided. The same mechanism can be also be used toinspect features of a specimen using illumination techniques that do notcause fluorescence. Further, in some embodiments, a pigment can be addedto a specimen and incoherent illumination techniques can be usedtargeted to the pigment to cause it to fluoresce. Inspection (sometimesreferred to as examination) refers to scanning, imaging, analyzing,measuring and any other suitable review of a specimen using thedisclosed incoherent microscopy inspection mechanism for fluorescenceimaging.

FIGS. 1A and 1B illustrate examples of fluorescence microscopyinspection systems using incoherent illumination for automaticallyanalyzing fluorescence emitted from a specimen (referred to herein as“FMIS 100”), according to some embodiments of the disclosed subjectmatter. At a high level, the basic components of FMIS 100, according tosome embodiments, include one or more illumination sources (e.g. lightsources 25, 25 a and 28) for providing incoherent light, a focusingmechanism 32 for finding the in-focus plane of a specimen, anilluminator 22, an imaging device 6, one or more objectives 35, a stage30, one or more filter mechanisms 15, a brightfield/darkfield slider 40and a control module 110 comprising hardware, software, and/or firmwareand a computer analysis system 115. As illustrated, control module 110,and computer analysis system 115 are coupled to inspection system 100via a communication channel 120. It is understood that communicationchannel 120 can include one or more signal transmitting means, such as abus, or wireless RF channel. It is also understood that FMIS 100 caninclude additional microscope components that are well known in the art.For example, FMIS 100 may include a frame (not depicted) to which thevarious components of FMIS 100 (e.g., one or more illumination sources,a focusing mechanism, an illuminator, an imaging device, one or moreobjectives, a stage, one or fore filter mechanisms, abrightfield/darkfield slider, a control module, a nosepiece, a beamsplitter) may be connected (e.g., for portability, stability, modularsupport, etc.). In some embodiments, a computer analysis system may beconnected to the frame and in some embodiments it may not. Othermicroscope components not listed herein, but well known in the art, canalso be connected to the frame.

FMIS 100 can be implemented as part of any suitable type of microscope.For example, in some embodiments, FMIS 100 can be implemented as part ofan optical microscope that uses reflected light (as shown in FIG. 1A)and/or transmitted light (as shown in FIG. 1B). More particularly, FMIS100 can be implemented as part of the nSpec® optical microscopeavailable from Nanotronics Imaging, Inc. of Cuyahoga Falls, Ohio.

In some embodiments, an XY translation stage can be used for stage 30.The XY translation stage can be driven by stepper motor, servo motor,linear motor, piezo motor, and/or any other suitable mechanism. The XYtranslation stage can be configured to move a specimen in the X axisand/or Y axis directions under the control of any suitable controller,in some embodiments.

In some embodiments, a focus mechanism 32 coupled to stage 30 can beused to adjust the stage in a Z direction towards and away fromobjective 35. Focus mechanism 32 can be used to make coarse focusadjustments of, for example, 0 to 5 mm, 0 to 10 mm, 0 to 30 mm, and/orany other suitable range(s) of distances. Focus mechanism 32 can also beused to move stage 30 up and down to allow specimens of differentthicknesses to be placed on the stage. Focus mechanism 32 can also beused in some embodiments to provide fine focus of, for example, 0 to 50μm, 0 to 100 μm, 0 to 200 μm, and/or any other suitable range(s) ofdistances. In some embodiments, focus mechanism 32 can also include alocation device. The location device can be configured to determine aposition of stage 30 at any suitable point in time. In some embodiments,any suitable position (e.g., the position of the stage when a specimenis in focus) can be stored in any suitable manner and later used tobring the stage back to that position, even upon reset and/or powercycling of FMIS 100. In some embodiments, the location device can be alinear encoder, a rotary encoder or any other suitable mechanism totrack the absolute position of stage 30 with respect to the objective.

According to some embodiments, FMIS 100 can include, one or moreobjectives 35. The objectives can have different magnification powersand/or be configured to operate with fluorescence, as well asbrightfield/darkfield, differential interference contrast (DIC),polarized light, cross-polarized light, and/or any other suitable formof illumination. The objective and/or illumination technique used toinspect a specimen can be controlled by software, hardware, and/orfirmware in some embodiments.

In some embodiments, a second focus mechanism (not shown) can be used todrive objective 35 in a Z direction towards and away from stage 30. Thesecond focus mechanism can be designed for coarse or fine focusadjustment of objective 35. The second focus mechanism can be a steppermotor, servo motor, linear actuator, piezo motor, and/or any othersuitable mechanism. For example, in some embodiments, a piezo motor canbe used and can drive the objective 0 to 50 micrometers (μm), 0 to 100μm, or 0 to 200 μm, and/or any other suitable range(s) of distances.

In some embodiments, communication between the control module (e.g., thecontroller and controller interface) and the components of FMIS 100 canuse any suitable communication technologies, that provide the ability tocommunicate with one or more other devices, and/or to transact data witha computer network. By way of example, implemented communicationtechnologies can include, but are not limited to: analog technologies(e.g., relay logic), digital technologies (e.g., RS232, ethernet, orwireless), network technologies (e.g., local area network (LAN), a widearea network (WAN), the Internet, Bluetooth technologies, Near-fieldcommunication technologies, Secure RF technologies, and/or any othersuitable communication technologies.

In some embodiments, operator inputs can be communicated to controlmodule 110 using any suitable input device (e.g., keyboard, mouse,joystick, touch, touch-screen, etc.).

In some embodiments, computer analysis system 115 can be coupled to, orincluded in, FMIS 100 in any suitable manner using any suitablecommunication technology, such as analog technologies (e.g., relaylogic), digital technologies (e.g., RS232, ethernet, or wireless),network technologies (e.g., local area network (LAN), a wide areanetwork (WAN), the Internet) Bluetooth technologies, Near-fieldcommunication technologies, Secure RF technologies, and/or any othersuitable communication technologies. Computer analysis system 115, andthe modules within computer analysis system 115, can be configured toperform a number of functions described further herein using imagesoutput by FMIS 100 and/or stored by computer readable media.

Computer analysis system 115 can include any suitable hardware (whichcan execute software in some embodiments), such as, for example,computers, microprocessors, microcontrollers, application specificintegrated circuits (ASICs), field-programmable gate arrays (FPGAs), anddigital signal processors (DSPs) (any of which can be referred to as ahardware processor), encoders, circuitry to read encoders, memorydevices (including one or more EPROMS, one or more EEPROMs, dynamicrandom access memory (“DRAM”), static random access memory (“SRAM”),and/or flash memory), and/or any other suitable hardware elements.

Computer-readable media can be any non-transitory media that can beaccessed by the computer and includes both volatile and nonvolatilemedia, removable and non-removable media. By way of example, and notlimitation, computer readable media can comprise computer storage mediaand communication media. Computer storage media can include volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information such as computer readableinstructions, data structures, program modules or other data. Computerstorage media includes, but is not limited to, RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital video disk (DVD) orother optical disk storage, magnetic cassettes, magnetic tape, magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store the desired information and which can beaccessed by the computer.

FMIS 100 can include one or more illumination sources, for example lightsources 25, 25 a and 28. In some embodiments, as shown for example inFIG. 1A, reflected illumination can be used (i.e., light originatingfrom above the specimen). Reflected light passes through verticalilluminator 22 to beam splitter 20. Beam splitter 20 can reflect thelight coming from the illumination source(s) at 90° downwards through anosepiece 23 and through brightfield channel 42 of objective 35 to thespecimen. In other embodiments, as shown for example in FIG. 1B,transmitted illumination can be used (i.e., light originating from belowthe specimen (light source 25 a)). The different illumination sourcescan be configured to provide illumination at wavelengths that aredifferent from each other. The different illumination sources can alsobe adjusted to control the intensity provided per unit area.

Beam splitter, as used herein, can refer to mirrors, dichroics, filtersor beam combiners that transmit light of a known, specified wavelengthand combines the transmitted light with light of another known,specified wavelength.

In some embodiments, as shown in FIG. 1B, in order to increase theintensity of light on a specimen, reflected light from illuminationsources 25/28 can be projected simultaneously with transmitted lightfrom illumination source 25 a. In some aspects, various illuminationsources can provide light at similar or equal wavelengths. In otherembodiments, FMIS 100 can include a single illumination source that canprovide light in ranges of varying wavelengths.

In some embodiments, for example, a first illumination source 25provides non-visible light 8 (e.g., projecting light with a wavelengthin the range of 10 to 400 nanometers (nm)), while the secondillumination source 28 provides visible light 9 (e.g., projecting lighthaving a wavelength in the range of 400 to 740 nanometers (nm)). Infurther embodiments, the illumination sources can provide other suitablewavelengths.

In some embodiments, as shown in FIGS. 1A and 1B, illumination source 25is positioned so that its light is projected in a substantiallyhorizontal direction towards vertical illuminator 22. Illuminationsources 25, 25 a and 28 can include a focusing lens suitable for thewavelength of the emitted light of each source.

In some embodiments that use two illumination sources, a beam splitter60 is placed in the optical pathway of both illumination sources (e.g.,illumination sources 25 and 28) before the light travels to verticalilluminator 22. The illumination sources can be activated so that theyare both providing illumination at the same time or at different times.Other placements of the illumination sources are contemplated, withoutdeparting from the scope of the disclosed technology. Note that acombination of the aforementioned devices, in any suitableconfiguration, can be used to reflect and transmit the desiredillumination sources and wavelengths. In some embodiments, a beamsplitter having a specific cut-off wavelength is selected in order toreflect the wavelengths of light emitted by illumination source 28 andto allow the wavelengths of light emitted from illumination source 25 topass through. Beam splitter 60 can be designed for a 45° angle ofincidence, so that rejected light from illumination source 28 isreflected at an angle of 90° and travels parallel to the light path fromillumination source 25. Other beam splitter designs are contemplated,without departing from the scope of the disclosed technology.

Note that, in some embodiments, any suitable incoherent illuminationsource(s) can be used with illumination sources 25, 25 a and 28,including, but not limited to, light-emitting diodes (LEDs), halogenlamps and/or fluorescent lights.

In some embodiments, a filter mechanism 15 can be used to allowspecified wavelength ranges from light sources 25 and 28 to pass throughto a specimen. Filter mechanism 15 (also referred to as an excitationfilter), can be, for example a slide having different bandpass filters(e.g., bandpass filters 16 and 17). Each bandpass filter allows certainwavelengths to pass through and blocks all other wavelengths. A motor ora mechanical mechanism can be used to select and position one of thebandpass filters. In other embodiments, a tunable filter that includessoftware, firmware, and/or hardware can be used to control the desiredwavelengths to pass through to the specimen. In some embodiments, thebandpass filter that is selected can be based on the bandgap propertiesof one or more of the materials in the specimen. By way of example, thebandpass filter may be selected to correspond with a wavelength energythat matches or exceeds the bandgap of one of the materials in aspecimen that is being inspected. In other words, the wavelength energythat is transmitted to a specimen, can be selected so that it causes atarget material within the specimen to fluoresce. Each material has aknown bandgap energy that is different from other materials. Bandgapenergy refers to the energy difference between the top of the valenceband and the bottom of the conduction band of a particular material.Fluorescence occurs when electrons in a material are excited bywavelengths of light, so that they absorb photons and emit an excitationlight (often the emitted light is emitted at a longer wavelength thanthe light absorbed). In addition to applying the appropriate wavelengthto excite a specimen, sufficient intensity must also be applied per unitarea, so that fluorescence can occur. The sufficiency of the intensityper unit area will depend on the material composition of the specimenand is generally in the range of 1 Watt/cm² through 11 Watt/cm². Forexample, an illumination source projecting light at a wavelength of 365nm and intensity of 4 Watts can be applied to a specimen of SiliconCarbide that has a bandgap energy of 3.26 eV to excite a fluorescenceresponse.

In further embodiments, the wavelength energy selected can correspond tothe wavelength energy needed to cause a target material within thespecimen and/or a pigment added to a target specimen to fluoresce. Note,that the term “excite” refers to the wavelength energy that causes aspecimen or a pigment added to the specimen to fluoresce (i.e., emitfluorescence).

In some embodiments, an excitation filter mechanism can be used based onthe desired microscopy inspection to be performed and allow, forexample, only wavelengths in a selected range to pass through. By way ofexample, the filter mechanism may be used to select wavelengths in thenon-visible range (e.g., ultraviolet light from illumination source 25)or wavelengths in the visible range (e.g., from illumination source 28)to pass through. In other embodiments, a filter mechanism can be used totransmit a specific wavelength of light to a specimen (e.g., thewavelength that corresponds to the bandgap of the material that is beinginspected and will excite the material).

Note that excitation filter slider 15 represents an example embodiment,and one or more excitation filter(s) can be placed at any suitableposition along the light path, before the light reaches the specimen. Insome embodiments, slider 40 can include an excitation filter and/or anexcitation filter can be included in nosepiece 23. These variousembodiments will be described herein.

In further embodiments, one or more emission filters can be used toallow the appropriate wavelengths to be transmitted from the specimen tothe imaging device, so that only the desired wavelengths are imaged.Similar to the excitation filter, the emission filter can be a bandpassfilter that allows certain wavelengths through and blocks others. Inother embodiments, a tunable filter that includes software, firmware,and/or hardware can be used to control the desired wavelengths that passthrough.

One or more emission filter(s) can be placed before each imaging device(e.g., emission filters 18 and 19 shown in FIG. 2), before tube lens 90(e.g., emission filter 21 shown in FIG. 1A), and/or in nose piece 23(e.g., emission filter F3 of filter slider 52 shown in FIG. 6C) totransmit the fluorescence response of a specimen. In some embodiments,an emission filter wheel can be used that further filters wavelengths ofcertain colors from reaching one or more imaging device. The emissionfilters, can be selected or controlled to allow specified wavelengths toreach the imaging devices. For example, to explore the fluorescenceresponse of Silicon Carbide at different wavelengths, different emissionbandpass filters (or a single wavelength) that allow different ranges ofwavelengths through (e.g., 414-440 nm, 500-550 nm or 590-670 nm) can beused. These filters can be applied one at a time, or if there aremultiple cameras, they can be applied simultaneously or as part of asequential slider (i.e., using a filter that allows a differentwavelength range in front of each imaging device).

FIG. 2 shows an example embodiment that includes two imaging devices 6and 7. In some embodiments, imaging devices 6 and 7 can be cameras thatincludes image sensors 5 and 4 respectively. Imaging devices 6 and 7 canbe used to capture images of a specimen. Image sensors 5 and 4 can be,for example, a CCD, a CMOS image sensor, and/or any other suitableelectronic device that converts light into one or more electricalsignals. Such electrical signals can be used to form images and/or video(including fluorescence images and/or video) of a specimen. In someembodiments, the imaging device can be a high quantum efficiency camerathat is effective at producing electronic charge from incident photons.In some embodiments, such electrical signals are transmitted for displayon a display screen connected to FMIS 100. In some embodiments, theimaging device can be replaced with or supplemented with an ocular or aneyepiece that is used to view a specimen, or with a spectrometer that isused to measure the spectral emissions from a specimen.

The imaging device can be positioned on a conjugate focal plane of FMIS100. In some embodiments, the imaging device can be mounted in otherlocations using appropriate components to adapt the selected location tothe optical characteristics of the system. In further embodiments, morethan one imaging device can be used. In some embodiments, the imagingdevice can be a rotatable camera that includes an image sensor,configured to allow the camera to be aligned to a specimen, a stageand/or a feature on a specimen. Some example methods for rotating acamera that can be used by FMIS 100 are described in U.S. Pat. No.10,048,477 entitled “Camera and Object Alignment to Facilitate LargeArea Imaging in Microscopy,” which is hereby incorporated by referenceherein in its entirety.

FIG. 2 includes emission filtering devices 18 and 19 that are eachcoupled to a respective imaging device. Each filtering device allowscertain wavelengths reflected off of/or emitted from the specimen to bereceived by the associated imaging device and blocks all otherwavelengths. FIG. 2 includes a beam splitter 24 that is positioned aboveilluminator 22 in the optical pathway of the light reflected offof/emitted from a specimen. The beam splitter can be positioned so thatwavelengths in a certain range are directed towards one imaging deviceand wavelengths of light in a different range are directed towards asecond imaging device.

Imaging of a specimen by FMIS 100 can be performed using various modesof observation including brightfield, darkfield, differentialinterference contrast (DIC), and others known to those familiar with theart.

In some embodiments, FMIS 100 can provide both brightfield and darkfieldillumination, either simultaneously or separately. Darkfieldillumination refers to an illumination technique that uses obliquelighting, rather than orthogonal light, to illuminate a sample. Anobjective can include an annular darkfield channel around thebrightfield channel that allows light to be transmitted to a sample atan angle of incidence of less than 90 degrees and greater than 0degrees, typically 25 to 80 degrees. In some embodiments, FMIS 100 caninclude a brightfield/darkfield slider 40 or other suitable mechanism(e.g., a cage cube) that allows only darkfield illumination, onlybrightfield illumination, a combination of brightfield/darkfieldillumination, or other types of illumination (e.g., DIC) to reach asample. Different configurations of brightfield/darkfield slider 40 willbe discussed in connection with FIGS. 2-5. In other embodiments,brightfield/darkfield illumination can be accomplished by coupling alight source above the darkfield channel and activating the light sourcevia control module 110 to provide darkfield illumination to a sample.Some example embodiments are discussed in connection with FIGS. 2-5.

In some embodiments, as shown in FIGS. 3A, 3B and 3C, FMIS 100 uses abrightfield/darkfield slider 40, a type of slider, that includes abrightfield configuration 43 and a darkfield configuration 44.Brightfield/darkfield slider 40 can be positioned anywhere along lightpath 10 that travel to a specimen (e.g., in the vertical illuminator,before beam splitter 20 or coupled above or below nosepiece 23).Brightfield/darkfield slider 40 includes two configurations: 43 and 44.In a first position, as shown in FIG. 3A, when configuration 43 ispositioned in the light path, the aperture in the center ofconfiguration 43 allows light 10 a to pass through and reflect off ofbeam splitter 20 through the brightfield channel in the center ofobjective 35 to provide brightfield illumination to a specimen, andblocks light from passing through to darkfield channel 41. Further, insome embodiments, the aperture in the center of configuration 43 can bereplaced with an excitation filter that allows only specific wavelengthsto reach a specimen (via reflection off of beam splitter 20).

In a second position, as shown in FIG. 3B, when configuration 44 ispositioned in the light path, the center aperture is closed, blockinglight from being transmitted to brightfield channel 42, and transmittinglight 10 b, via reflection off of beam splitter 20, through darkfieldchannel ring 41 to provide oblique illumination to a specimen. A motoror a mechanical mechanism can be used to select and position one of thebrightfield/darkfield slider configurations. Light 10 c reflected fromthe specimen then travels through objective 35 to the imaging device(s).

Other configurations of brightfield/darkfield slider 40 are possible, asshown for example in FIGS. 4A (brightfield/darkfield slider 40 a) and 4B(brightfield/darkfield slider 40 b). Brightfield/darkfield slider 40 acan include configurations 45 (which includes a ring of lights 46 (e.g.,an LED light ring) around a closed center) and 43 (described inconnection with FIGS. 3A-3C). When configuration 45 is positioned in thelight path, and LED lights 46 are activated, oblique illumination can betransmitted to a specimen via darkfield channel 41 (by reflecting off ofbeam splitter 20). Since the center of the ring is closed and blockslight from entering a brightfield channel (via reflection off of beamsplitter 20), no brightfield light is transmitted to a specimen.

As shown in FIG. 4B, brightfield/darkfield slider 40 b can includeconfigurations 47 (which includes an LED ring of lights 46 around anaperture) and 43 (described in connection with FIGS. 3A and 3B). Whenconfiguration 47 is positioned in the light path and LED ring of lights46 is activated, oblique illumination can be projected and transmittedto a specimen (via reflection off of beam splitter 20), whilesimultaneously brightfield illumination can pass through the aperture inthe center and be transmitted to a specimen through brightfield channel42 of objective 35 (as shown in FIG. 5B).

In some embodiments, as shown in FIGS. 5A (showing an example FMIS 100),5B (showing the details of an example nosepiece 23), 6A (showing anexample darkfield insert 51) and 6B (showing an example cylinder 29), acylinder 29 (also referred to herein as an “attachment” or a “cylinderattachment”) can be fastened to nosepiece 23 of FMIS 100 (e.g., viascrews or other fasteners), and an objective 35 (including annulardarkfield channel 41 and brightfield channel 42) can be fastened tocylinder 29, above darkfield channel 41. Further, a darkfield insert 51having a ring of lights 46 (e.g., LED lights 46) can be fastened tocylinder 29, above darkfield channel 41. Such a configuration allowscylinder 29 to be fastened into any nosepiece and to be used with anyobjective. Note, cylinder 29 can be any suitable shape. Further, thering of lights included on darkfield insert 51 can include any suitablelight that emits one or more wavelengths, and can be flexiblyinterchanged with another insert 51 having a ring of lights thatincludes a different type of light and emits a different wavelength (orset of wavelengths).

In some embodiments, a filter slider 52, a type of slider, with multipleemission/excitation filters F1, F2, F3 . . . FN can be coupled tocylinder 29, below darkfield insert 51. In some embodiments, filter F1of slider 52 includes an aperture that allows the light from lights 46,when activated, to pass through unfiltered via darkfield channel 41 to aspecimen. Filter F2 of slider 52 includes an excitation filter thatallows only certain darkfield and brightfield wavelengths to reach aspecimen. In some embodiments, the excitation filter can include anaperture in the center, and only filter the darkfield light that reachesa specimen. In further embodiments, filter F3 can include differentfilters for the brightfield and darkfield channels. For example, thedarkfield filter can be configured to filter the darkfield excitationlight, while the brightfield filter can be configured as an emissionfilter to filter the light emitted from a specimen before it reaches oneor more imaging devices. Slider 52 can include other suitable filtersthat target specific excitation wavelengths from reaching a specimenand/or target specific emission wavelengths from reaching one or moreimaging devices.

Note that a combination of the aforementioned excitation and emissionfilters, in any suitable configuration, can be used to reflect andtransmit the desired illumination sources and wavelengths.

FIG. 7A shows at a high level, an example method 700 for illuminating aspecimen using a FM inspection system to achieve desired spectralemissions and other desired illumination for image capture, inaccordance with some embodiments of the disclosed subject matter. Insome embodiments, method 700 can use FMIS 100.

At 710, a specimen to be examined can be placed on specimen stage 30. Insome embodiments, the specimen is brought into focus before the lightsources and filters of FMIS 100 are selected.

At 720, the settings of FMIS 100 can be adjusted for image capture. Thiscan be performed manually or auotmatically (e.g., using a computeralgorithm) based on, for example, the features of a specimen beingexamined, or the material composition of a specimen. In someembodiments, control module 110 can activate and adjust the wavelengthsand intensity of light from the light source(s), as well as thecorresponding excitation and emission filters, according to storedinformation for a particular specimen, specimen class and/or any othersuitable classification group. The stored information can include a mapthat identifies the type and location of known features on a specimen(“specimen feature map” or “feature map”). The stored information canalso include the material composition of a specimen, the optimal FMinspection system settings for capturing different images of thespecimen at different regions of interest (e.g., by specifying thewavelength and intensity of light to direct at a specimen, by selectingand adjusting the appropriate excitation and/or emission filters).Further, the stored information can include information on type andlocation of known or expected defects of a specimen. Methods forselecting suitable stored information are further discussed inconnection with FIG. 8.

At 730, according to some embodiments, one or more images of a specimenare captured by FMIS 100. Steps 720 and 730 can be repeated as manytimes as desired to capture different images of the specimen. Forexample, adjustments can be made to the intensity and wavelengths oflight sources 25, 25 a and/or 28 and corresponding excitation andemission filters to capture different images of the specimen.Adjustments to the light sources can be made, for example, based onstored information for a specimen, including specimen composition, knownor expected defects of the specimen, and/or a specimen feature map.Further, wavelengths of light sources 25, 25 a and/or 28 andcorresponding filters can be adjusted for different regions of interestof the specimen (as indicated by a specimen feature map or otherwise),and images can be captured for each region of interest. In someembodiments, wavelengths of light sources 25, 25 a and/or 28 andcorresponding filters can be selected in a range appropriate to providedesired excitation to the specimen and/or region of interest. Further,different images of a specimen can be captured by adjusting the type ofillumination provided to the specimen, such as, applying brightfield,darkfield, a combination of brightfield and darkfield, and/or DICillumination.

FIG. 7B illustrates steps of an example process 705 for identifying aspecimen classification and automatically adjusting the light sourcesand filters for FMIS 100, according to some aspects of the disclosedtechnology. Process 705 begins with step 740 which image data isreceived, for example, by an image processing system e.g., imageprocessing module 834 (as shown in FIG. 8). In some approaches, theimage data can be included in a received image of a specimen that istaken by an imaging device, as part of FMIS 100. The image data caninclude all or a portion of a specimen that is disposed on a stage ofFMIS 100.

In step 750, the image data is analyzed to identify a classification ofthe specimen. In some instances image analysis may be performed toidentify a subset of the specimen, such as a particular region, featureor material within the specimen. As discussed below, machine learningclassifiers, computer vision, and/or artificial intelligence can be usedto identify/classify the specimen and features on a specimen. An exampleclassification method using convolutional neural networks (CNNs) isshown in FIG. 10.

Subsequently, stored information can be automatically selected based onthe specimen (or feature) classification (step 760). Thespecimen/feature classification can be used to query a database (e.g.,stored information database 836) that contains stored informationassociated with: a specimen, the material composition of a specimen,specimen feature types, and/or other suitable classification group. Byreferencing the specimen classification determined in step 750, storedinformation appropriate for the specimen can be automatically identifiedand retrieved. As discussed above, the stored information can contain avariety of settings data that describe configurations of FMIS 100 thatcan be used to achieve the optimal illumination and image capture forthe specimen, feature, and/or material being observed.

FIG. 7C illustrates steps of an example process 710 for automaticallyidentifying and/or classifying specimen defects, according to someaspects of the disclosed technology. In some implementations, step 780can follow step 770 discussed above with respect to FIG. 7B. However, itis understood that process 710 can be performed independently of thevarious steps of processes 700, and 705, discussed above.

In step 780, image data, including fluorescence, is received from theimaging device following application of adjustments/settings of the FMinspection system. In some approaches, step 780 can be performedsubsequent to the automatic classification of the specimen, as describein step 770, above. As such, the image data received in step 780 canrepresent an image of the specimen taken under optimized or improvedlighting conditions, as realized by the settings selection performed forthe FM inspection system.

In step 785, the new image data, including fluorescence, is provided toa defect detection classifier that is configured to automaticallyidentify/detect and/or classify defects/features of the specimen.Defect/feature detection and classification can be performed withoutknowledge of the specimen classification or type. However, in someembodiments, a specimen classification and/or associated storedinformation can be used as inputs to the defect detection classifier,and thereby used to inform the process of defect/feature detection andidentification.

In step 790, one or more defects/features of the specimen are identifiedand/or classified. The process of identifying and/or classifyingspecimen defects/features can be carried out in different ways,depending on the desired implementation. For example, defect/featureidentification can be used to automatically generate or update a featuremap and/or stored information associated with the given specimen and/orspecimen classification (step 795). As such, identification of noveldefects/features can be used to improve (train) future defect/featureclassification calculations, as well as to improve the automated processof adjusting FM inspection system settings, as described in process 705.In some aspects, defect/feature identification and/or classification canbe used to trigger an alert, for example to notify a user of the FMinspection system as to the existence of the detected defect/featureand/or defect/feature type (classification).

It is understood that at least some of the portions of methods 700, 705and 710 described herein can be performed in any order or sequence notlimited to the order and sequence shown in and described in connectionwith FIGS. 7A, 7B and 7C, in some embodiments. Also, some portions ofprocesses 700, 705 and 710 described herein can be performedsubstantially simultaneously where appropriate or in parallel in someembodiments. Additionally, or alternatively, some portions of process700, 705 and 710 can be omitted in some embodiments. Methods 700, 705and 710 can be implemented in any suitable hardware and/or software. Forexample, in some embodiments, methods 700, 705 and 710 can beimplemented in FM inspection system 100.

FIG. 8 shows the general configuration of an embodiment of computeranalysis system 115, in accordance with some embodiments of thedisclosed subject matter. Although computer analysis system 115 isillustrated as a localized computing system in which various componentsare coupled via a bus 805, it is understood that various components andfunctional computational units (modules) can be implemented as separatephysical or virtual systems. For example, one or more components and/ormodules can be implemented in physically separate and remote devices,such as, using virtual processes (e.g., virtual machines or containers)instantiated in a cloud environment.

Computer analysis system 115 includes a processing unit (e.g., CPU/sand/or processor/s) 810 and bus 805 that couples various systemcomponents including system memory 815, such as read only memory (ROM)820 and random access memory (RAM) 825, to processor/s 810.

Memory 815 can include various memory types with different performancecharacteristics, such as memory cache 812. Processor 810 is coupled tostorage device 830, which is configured to store software andinstructions necessary for implementing one or more functional modulesand/or database systems, such as stored information database 836. Eachof these modules and/or database systems can be configured to controlprocessor 810 as well as a special-purpose processor where softwareinstructions are incorporated into the actual processor design. As such,image processing module 834 and the stored information database 836 canbe completely self-contained systems. For example, imagine processingmodule 834 can be implemented as a discrete image processing system,without departing from the scope of the disclosed technology.

To enable user interaction with computer analysis system 115, inputdevice 845 can represent any number of input mechanisms, such as amicrophone for speech, a touch-sensitive screen for gesture or graphicalinput, keyboard, mouse, motion input and so forth. An output device 835can also be one or more of a number of output mechanisms known to thoseof skill in the art. In some instances, multimodal systems can enable auser to provide multiple types of input to communicate with computeranalysis system 115, for example, to convey specimen informationrelating to a specimen type/classification, or other characteristics.Communications interface 840 can generally govern and manage the userinput and system output. There is no restriction on operating on anyparticular hardware arrangement and therefore the basic features heremay easily be substituted for improved hardware or firmware arrangementsas they are developed.

Storage device 830 is a non-transitory memory and can be a hard disk orother types of computer readable media that can store data accessible bya computer, such as magnetic cassettes, flash memory cards, solid statememory devices, digital versatile disks, cartridges, random accessmemories (RAMs) 825, read only memory (ROM) 820, and hybrids thereof.

In practice, stored information database 836 can be configured toreceive, store and update context data associated with a specimen, aspecimen class and/or or other suitable specimen classification. Contextdata for each specimen/specimen class/specimen classification caninclude, but is not limited to: a computer aided design (CAD) file of aspecimen and/or features of a specimen, a feature map identifyingfeatures and their locations, images of specimens/features of specimenscaptured by FMIS 100, images of known specimens and/or features, knowndimensions, material composition, mechanical and/or physical propertiesof a specimen, spectral variation maps for known materials or specimens,common stacking faults, structural defects or other defects associatedwith a specimen, optimal FM inspection settings for features of aspecimen, a specimen or specimen classification, identification ofregions of interest/or materials of interest to examine. In someembodiments regions of interest can be identified on a feature map.Storage information database 836 can be coupled to image processingmodule 834 and can transmit data to and receive data from imageprocessing module 834. Further, context data can include data related tothe FMIS 100 being used to inspect a specimen such as: the number oflight sources, the wavelength range and intensity for each light source,the number of imaging devices and the different types ofexcitation/emission filters and their locations, of FMIS 100; the rangeof possible distances between specimen stage 30 and objective 35.

Processor 810 can include an image processing module 834. Imageprocessing module 834 can be used in conjunction with stored informationdatabase 836 to classify a specimen based on: image data, includingfluorescence, received in a specimen image(s); context data retrievedfrom stored information database 836, and/or other received specimencharacteristics, such as those manually provided by a user, for example,via input 845. Additionally, image processing module can be configuredto classify specific specimen features, determine other physical and/ormechanical specimen properties (e.g., specimen reflectivity, specimendimensions, specimen material composition). Classifications of specimentypes, and specimen features/properties can be stored in storedinformation database 836.

In some embodiments, once a specimen type, specific features and/or thematerial composition of a specimen has been determined (e.g., by imageprocessing module 834), additional context data associated with thedetermined specimen type/features can be retrieved from storedinformation database 836 and sent to control module module 110 to adjustthe settings of the FMIS 100 to capture specific specimen images and/orto guide the inspection of the specimen by FMIS 100 (e.g., by capturingimages of specific features and/or regions of interest).

In some embodiments, an image processing module 834 can receive anentire specimen scan, or one or more images of a specimen. Imageprocessing module 834, as shown in FIG. 9, can apply one or moreartificial intelligence algorithm(s) to classify a specimen type, aswell as features on the specimen.

As understood by those of skill in the art, artificialintelligence/machine learning based classification techniques can varydepending on the desired implementation, without departing from thedisclosed technology. For example, machine learning classificationschemes can utilize one or more of the following, alone or incombination: hidden Markov models; recurrent neural networks;convolutional neural networks (CNNs); deep learning; Bayesian symbolicmethods; general adversarial networks; support vector machines; imageregistration methods; applicable rule-based system. Where regressionalgorithms are used, they may include including but are not limited to:a Stochastic Gradient Descent Regressor, and/or a Passive AggressiveRegressor, etc.

Machine learning classification models can also be based on clusteringalgorithms (e.g., a Mini-batch K-means clustering algorithm), arecommendation algorithm (e.g., a Miniwise Hashing algorithm, orEuclidean LSH algorithm), and/or an anomaly detection algorithm, such asa Local outlier factor. Additionally, machine learning models can employa dimensionality reduction approach, such as, one or more of: aMini-batch Dictionary Learning algorithm, an Incremental PrincipalComponent Analysis (PCA) algorithm, a Latent Dirichlet Allocationalgorithm, and/or a Mini-batch K-means algorithm, etc.

In some instances, machine learning models can be used to performclassification of specimens, materials within a specimen, specimenfeatures, and/or other specimen characteristics. In some aspects, imagedata from specimen images can be provided as an input to a machinelearning classification system, for example, by image processing module834. Classifier output can specify a sample or feature classificationthat can then be used to identify specific regions of interest on aspecimen for further inspection by FMIS 100, and to provide instructionsto control module 110 of FMIS 100 on the type of lights sources andfilters that should be used to inspect those areas of interest.

Such algorithms, networks, machines and systems provide examples ofstructures used with respect to any “means for determining a feature ofa specimen using artificial intelligence” or “means for determining aregion of interest of a specimen for further inspection using artificialintelligence” or “means for determining a feature of a specimen usingartificial intelligence.”

Further, for each feature on the specimen or for a region of interest,the image processing module can apply one or more artificialintelligence algorithm(s) to: i) detect the feature; ii) classify thefeature type; iii) determine location of the feature on the specimen;iv) determine the material composition of the specimen/feature; v)determine optimal settings for FMIS 100 (e.g., the wavelength excitationsetting, the wavelength emission setting, the illumination techniqueapplied). to inspect a feature/specimen/material. In some embodiments,the algorithm(s) used by image processing module 834 can considercontext date like location of the feature on a specimen, the type ofspecimen being inspected, the physical and mechanical properties of thespecimen being inspected, similar features on the same or similarspecimen, a reference feature map for the inspected specimen, the FMinspection system settings used to generate the specimen scan orspecimen image.

Examples of machine-learning artificial intelligence based imageprocessing algorithm that can be used by image processing module 834 isimage registration as described by: Barbara Zitova, “Image RegistrationMethods: A Survey,” Image and Vision Computing, Oct. 11, 2003, Volume21, Issue 11, pp. 977-1000, which is hereby incorporated by referenceherein in its entirety. The disclosed methods are just examples and arenot intended to be limiting. By way of example,machine-learning/artificial intelligence models can be trained usingmultiple sources of training data, including, but not limited to: acomputer aided design (CAD) file of a specimen and/or features of aspecimen, a specimen feature map identifying features and theirlocations on a specimen, images of known specimens and/or features,and/or information about known specimens (e.g., a specimen's dimensions,a speciman's material composition, the mechanical and/or physicalproperties of a specimen, spectral variation maps for known materials orspecimens, common stacking faults, structural defects, feature maps thatidentify where features within a specimen classification are commonlylocated).

In some embodiments, as shown in FIG. 9, an image processing algorithm905 is first trained with training data 920 so that image processingmodule 834 can recognize and classify a specimen, and detect andrecognize features on a specimen. Multiple training techniques may beused and may depend upon the particular classifier model being used. Inone example, a CNN, such as a 13-layer CNN, etc., may be trained overmultiple epochs using stochastic gradient descent to explore arespective error space. In one example, 80 epochs are used for trainingand the stochastic gradient descent can include a momentum factor.Additionally, an adaptive learning rate can be used such as, for exampleand without imputing limitation, an adjustment to the learning rate from0.1 (e.g., as a step value in the stochastic gradient descent) duringearly epochs to 0.01 in later epochs.

Training data 920 can include labeled examples of known types ofspecimens and features. For each classification being trained for (e.g.,feature, feature type, type of defect, etc.), training data 920 canfurther include labeled imaged of deformed features (these can be actualdeformed features or deformed features that were simulated according topredefined parameters) and training data 920 can include labeled imagesof such deformed features. Training data 920 can also include labeledimages of each defect type rotated from 0-360 degrees. Training data 920can also include labeled images of each defect type generated atdifferent sizes. One example of training data 920 are images includinglabeled stacking faults having different structures, shapes and sizes,and the corresponding fluorescence emission for each type of stackingfault. Further, the labeled images can also include additional contextdata like information specifying the settings for FMIS 100 (e.g.,wavelength excitation setting, wavelength emission setting, lightingtechnique applied), the material composition of a feature or a specimen,location of a feature on a specimen, physical/mechanical properties ofthe feature and/or any other suitable characteristic. In someembodiments, training data can also include unlabeled data.

Once the image processing algorithm is trained it can be applied byimage processing module 834 to a received specimen scan(s) or image(s)of a specimen to classify specimen type, detect features, classify faulttype, determine feature and/or fault locations, determine specimencomposition, and determine optimal FM inspection system settings fordetecting a feature/specimen. The output data can be displayed visually,printed, or generated in file form and stored in database 836 ortransmitted to other components for further processing.

In some embodiments, output data can be sent to a feature map generatormodule 832 to generate a feature map for the specimen. In someembodiments, the output data may comprise multiple images. The generatedfeature map can identify and locate features on the specimen. Thegenerated feature map can be displayed visually, printed, or generatedin file form and stored in stored information database 836 ortransmitted to other modules for further processing.

Further, the generated feature map can be used to focus furtherinspection by FMIS 100 on specific features and/or regions of aspecimen. Based on the characteristics of the features and regions,stored information can be retrieved from stored information database836. For example, for each feature and/or region of interest,instructions can be retrieved from stored information database 836 forapplying different lights sources and illumination techniques, atdifferent wavelengths and intensity levels, using differentexcitation/emission filters to capture different images and transmittedto control module 110. For example, by applying different bandpassemission filters before one or more imaging device, differentfluorescence emissions can be detected and different features of aspecimen identified (e.g., irregularities or defects in the surface) ofa specimen.

FIG. 10 describes one embodiment for training image processing module834 that uses a deep convolution network classifier 1005. Classifier1005 can be trained using simulated augmented data 1007. For example,known defects for different types of specimens can be generated atdifferent orientations, different sizes, different pixel intensities,different locations on a specimen (1006 and 1009). The shapes of theseknown defects can be blurred and/or distorted. Once trained, one or morecandidate images of FMIS 100 can be input into a classifier (1009). Insome embodiments, the image (1001) is first processed by detectingcertain regions and extracting features from those regions (1002 and1003). Classifier 1005 is then used to analyze the extracted featuresand to classify the features into types and locate those features on thespecimen (1010). Note, some example methods for locating a feature onthe specimen that can be used by FMIS 100 are described in U.S. patentapplication Ser. No. 16/262,017 entitled “Macro Inspection Systems,Apparatus and Methods,” which is hereby incorporated by reference hereinin its entirety. In some embodiments, the known defects include stackingfaults having different structures, sizes and shapes.

In some embodiments, any suitable computer readable media can be usedfor storing instructions for performing the functions and/or processesdescribed herein. For example, in some embodiments, computer readablemedia can be transitory or non-transitory. For example, non-transitorycomputer readable media can include media such as non-transitorymagnetic media (such as hard disks, floppy disks, etc.), non-transitoryoptical media (such as compact discs, digital video discs, Blu-raydiscs, etc.), non-transitory semiconductor media (such as flash memory,electrically programmable read only memory (EPROM), electricallyerasable programmable read only memory (EEPROM), etc.), any suitablemedia that is not fleeting or devoid of any semblance of permanenceduring transmission, and/or any suitable tangible media. As anotherexample, transitory computer readable media can include signals onnetworks, in wires, conductors, optical fibers, circuits, and anysuitable media that is fleeting and devoid of any semblance ofpermanence during transmission, and/or any suitable intangible media.

The various systems, methods, and computer readable media describedherein can be implemented as part of a cloud network environment. Asused in this paper, a cloud-based computing system is a system thatprovides virtualized computing resources, software and/or information toclient devices. The computing resources, software and/or information canbe virtualized by maintaining centralized services and resources thatthe edge devices can access over a communication interface, such as anetwork. The cloud can provide various cloud computing services viacloud elements, such as software as a service (SaaS) (e.g.,collaboration services, email services, enterprise resource planningservices, content services, communication services, etc.),infrastructure as a service (IaaS) (e.g., security services, networkingservices, systems management services, etc.), platform as a service(PaaS) (e.g., web services, streaming services, application developmentservices, etc.), and other types of services such as desktop as aservice (DaaS), information technology management as a service (ITaaS),managed software as a service (MSaaS), mobile backend as a service(MBaaS), etc.

The provision of the examples described herein (as well as clausesphrased as “such as,” “e.g.,” “including,” and the like) should not beinterpreted as limiting the claimed subject matter to the specificexamples; rather, the examples are intended to illustrate only some ofmany possible aspects. A person of ordinary skill in the art wouldunderstand that the term mechanism can encompass hardware, software,firmware, or any suitable combination thereof.

Unless specifically stated otherwise as apparent from the abovediscussion, it is appreciated that throughout the description,discussions utilizing terms such as “determining,” “providing,”“identifying,” “comparing” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system memories or registersor other such information storage, transmission or display devices.Certain aspects of the present disclosure include process steps andinstructions described herein in the form of an algorithm. It should benoted that the process steps and instructions of the present disclosurecould be embodied in software, firmware or hardware, and when embodiedin software, could be downloaded to reside on and be operated fromdifferent platforms used by real time network operating systems.

The present disclosure also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored on acomputer readable medium that can be accessed by the computer. Such acomputer program may be stored in a computer readable storage medium,such as, but is not limited to, any type of disk including floppy disks,optical disks, CD-ROMs, magnetic-optical disks, read-only memories(ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic oroptical cards, application specific integrated circuits (ASICs), or anytype of non-transient computer-readable storage medium suitable forstoring electronic instructions. Furthermore, the computers referred toin the specification may include a single processor or may bearchitectures employing multiple processor designs for increasedcomputing capability.

The algorithms and operations presented herein are not inherentlyrelated to any particular computer or other apparatus. Variousgeneral-purpose systems may also be used with programs in accordancewith the teachings herein, or it may prove convenient to construct morespecialized apparatus to perform the required method steps andsystem-related actions. The required structure for a variety of thesesystems will be apparent to those of skill in the art, along withequivalent variations. In addition, the present disclosure is notdescribed with reference to any particular programming language. It isappreciated that a variety of programming languages may be used toimplement the teachings of the present disclosure as described herein,and any references to specific languages are provided for disclosure ofenablement and best mode of the present disclosure.

The FM inspection apparatus, method and system have been described indetail with specific reference to these illustrated embodiments. It willbe apparent, however, that various modifications and changes can be madewithin the spirit and scope of the disclosure as described in theforegoing specification, and such modifications and changes are to beconsidered equivalents and part of this disclosure.

We claim:
 1. A microscopy system comprising: a frame; one or moreincoherent light sources connected to the frame and configured to emitat least a first wavelength of light that will cause a specimen tofluoresce and a second wavelength of light that will not cause aspecimen to fluoresce, wherein the emitted light is configured to bedirected to the specimen; an excitation filter connected to the frameand configured to filter light from the one or more incoherent lightsources, wherein the filtered light is configured to match apredetermined bandgap energy associated with the specimen; an objectiveconnected to the frame, the objective comprising a brightfield channeland a darkfield channel; a nosepiece connected to the objective via anattachment; a slider connected to the frame and positioned along alightpath between the objective and the one or more incoherent lightsources, wherein the slider includes at least one configurationconfigured to transmit light along the lightpath to at least a darkfieldchannel configured to direct light to the specimen at an oblique angle;an emission filter connected to the frame and configured for filteringselected wavelengths of light reflected from the specimen to a receivingcamera; one or more processors; and a memory storing instructions that,as a result of being executed by the one or more processors, cause themicroscopy system to: obtain image data from the receiving camera, theimage data based on the directed light reflected from the specimen;classify the specimen with a trained classifier based on the receivedimage data; retrieve stored system configurations associated with theclassification of the specimen; apply the system configurations to oneor more of the incoherent light sources, excitation filter, emissionfilter, or the receiving camera; obtain additional image data from thereceiving camera, the additional image data obtained after the systemconfigurations have been applied; identify specimen defects with animage data model based on the obtained additional image data; andgenerate a feature map based on the specimen defects.
 2. The system ofclaim 1, wherein the at least one configuration of the slider isconfigured to transmit light to both the brightfield channel and thedarkfield channel.
 3. The system of claim 1, wherein the slider is: afilter slider connected to the frame and positioned below the darkfieldinsert, the filter slider configured to provide multiple types ofexcitation filters; and one or more additional emission filters for oneor more of the brightfield channel or the darkfield channel.
 4. Thesystem of claim 1, further comprising at least a second camera andwherein the emitted light comprises visible light and non-visible lightdirected to respective cameras.
 5. The system of claim 1, furthercomprising one or more additional cameras connected to the frame, eachadditional camera configured to receive respective unique wavelengths oflight.
 6. The system of claim 1, further comprising: one or moreprocessors; and a memory storing instructions that, when executed by theone or more processors, cause the one or more processors to: receiveimage data from the receiving camera, the image data based on thedirected light from the specimen; classify the specimen with a trainedclassifier based on the received image data; retrieve stored systemconfigurations associated with the classification of the specimen; andapply the system configurations to one or more of the light sources,excitation filter, emission filter, or receiving camera.
 7. The systemof claim 6, wherein the memory stores further instructions to: receiveadditional image data from the receiving camera, the additional imagedata received after the system configurations have been applied;identify specimen defects with an image data model based on the receivedadditional image data; and generate a feature map based on the specimendefects.
 8. The system of claim 1, wherein the one or more incoherentlight sources further comprises: a first light source connected to theframe and configured to emit reflected light from the one or moreincoherent light sources to the specimen; and an additional light sourceattached to the frame below the specimen and configured to increase theintensity of light on the specimen by emitting light directed at thespecimen simultaneously to the light emitted by the one or firstincoherent light source.
 9. The system of claim 1, further comprising abeam splitter connected to the frame and configured to direct emittedlight towards the specimen.
 10. A method comprising: emitting from oneor more incoherent light sources at least a first wavelength of lightthat causes a specimen to fluoresce and a second wavelength of lightthat does not cause the specimen to fluoresce, wherein the emitted lightis directed to the specimen; filtering the emitted light through anexcitation filter, the filtered light matching a predetermined bandgapenergy; transmitting the emitted light through a slider to the specimenvia a darkfield channel of an objective at an oblique angle; directing,using an emission filter, light reflected from the specimen to areceiving camera, the reflected light in response to the directedfiltered light, wherein the directed light reflected from the specimencomprises selected wavelengths; obtaining, from the receiving camera,image data based on the directed light reflected from the specimen;classifying the specimen with a trained classifier based on the imagedata; retrieving stored system configurations associated withclassification of the specimen; applying the system configurations toone or more of the incoherent light sources, the excitation filter, theemission filter, or the receiving camera; obtaining additional imagedata from the receiving camera, the additional image data obtained afterthe system configurations are applied; identifying specimen defects withan image data model based on the obtained additional image data; andgenerating a feature map based on the specimen defects.
 11. The methodof claim 10, further comprising transmitting the filtered light to thespecimen through the slider to a brightfield channel of the objective.12. The method of claim 10, wherein a darkfield insert, positioned abovethe darkfield channel of the objective, comprising a ring of lights,emits light to the specimen at an oblique angle via the darkfieldchannel of the objective.
 13. The method of claim 10, wherein theemitted light comprises visible light and non-visible light, and furthercomprising receiving, by a second camera, at least a portion of thedirected light reflected from the specimen.
 14. The method of claim 10,further comprises receiving, by one or more additional cameras, uniquewavelengths of the light reflected from the specimen.
 15. The method ofclaim 10, further comprising: receiving image data from the receivingcamera, the image data based on the directed light reflected from thespecimen; classifying the specimen with a trained classifier based onthe received image data; retrieving stored system configurationsassociated with the classification of the specimen; and applying thesystem configurations to one or more of the light sources, excitationfilter, emission filter, or receiving camera.
 16. The method of claim15, further comprising: receiving additional image data from thereceiving camera, the additional image data received after the systemconfigurations have been applied; identifying specimen defects with animage data model based on the received additional image data; andgenerating a feature map based on the specimen defects.
 17. The methodof claim 10, further comprising: emitting a first light from a firstlight source of the one or more incoherent light sources toward thespecimen; and increasing the intensity of light on the specimen, byemitting a second light directed at the specimen from an additionallight source of the one or more incoherent light sources, from below thespecimen, wherein the additional light is emitted simultaneously to thelight emitted by the first light source of the one or more incoherentlight sources.
 18. The method of claim 10, further comprising directingthe emitted light towards the specimen with a beam splitter.
 19. Amicroscopy apparatus comprising: one or more incoherent light sourcesconfigured to emit at least a first wavelength of light that will causea specimen to fluoresce and a second wavelength of light that will notcause the specimen to fluoresce, wherein the emitted light is configuredto be directed to the specimen; an excitation filter configured tofilter light from the one or more light sources, wherein the filteredlight is configured to match a predetermined bandgap energy associatedwith the specimen; an objective comprising a brightfield channel and adarkfield channel; a nosepiece connected to the objective via anattachment; a darkfield insert fastened to the attachment and positionedabove the darkfield channel of the objective, the darkfield insertcomprising a ring of lights configured to project light at the specimenat an oblique angle; an emission filter configured for filteringselected wavelengths of reflected light from the specimen to a receivingcamera; one or more processors; and a memory storing instructions that,when executed by the one or more processors, cause the one or moreprocessors to: receive image data from the receiving camera, the imagedata based on the directed light reflected from the specimen; classifythe specimen with a trained classifier based on the received image data;retrieve stored system configurations associated with the classificationof the specimen; apply the system configurations to one or more of thelight sources, excitation filter, emission filter, or receiving camera;receive additional image data from the receiving camera, the additionalimage data received after the system configurations have been applied;identify specimen defects with an image data model based on the receivedadditional image data; and generate a feature map based on the specimendefects.